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The purpose of this technical report is to summarize the research methodology of the 
Novice Teacher Study (NTS) strand of the Teacher Quality Partnership (TQP) research. The 
report is divided into three sections: a brief description of TQP; a detailed description of the 
NTS research design and methodology; and concluding remarks related to the challenges of 
conducting field research as part of a large-scale research initiative. In preparing this 
technical report, we drew from artifacts documenting the research processes; from regional 
and national presentations; and from materials distributed to external audiences. Preliminary 
planning for the NTS began in 2004, but this report focuses on 2005-06 (pilot year) and 
2006-08 (years 1 and 2) 1 . 

THE TEACHER QUALITY PARTNERSHIP RESEARCH 

The NTS is one strand of related research under the umbrella of the Ohio Teacher 
Quality Partnership (formerly the Ohio Partnership of Accountability). Lasley, Siedentop, 
and Yinger (2006) describe the beginnings of TQP: 

The Ohio TQP has embarked on a series of research studies to learn more 
about the characteristics of effective teachers and to identify the patterns of 
teacher performance in both novice and experienced teachers that commingle 
to enhance student achievement at different grade levels, in different subjects, 
and with different types of students, (p. 16) 

TQP encompasses a consortium of all 50 teacher preparation institutions in Ohio and 
partnerships with the Ohio Department of Education and the Ohio Board of Regents. A board 
of directors providing leadership for TQP includes education deans from the University of 
Cincinnati, Cleveland State University, the University of Dayton, and The Ohio State 
University. An external audit panel comprising national experts in educational research 
provides critical feedback to each strand of the TQP research. An Ohio advisory board with 
representatives from all educational stakeholder groups in Ohio monitors project goals and 
planning and will become the forum for exploring the policy implications of the research 
findings. 

Yinger (2005) summarized the TQP research in a document prepared for external 
funders: “The focus of the TQP project is to better understand the pathways and influences 
related to the teacher and to his or her ultimate influence on students’ academic performance” 
(p. 8). To establish the connection between teacher influence and student academic 
performance, the project has incorporated a value-added modeling (VAM) measure of 
teacher effect on student learning. References to high value-added teachers (HVATs) 



The Novice Teacher Study evolved with the other field study strand of TQP, the Experienced Teacher Study. 
Key personnel contributing to the NTS include Dr. Katie Kinnucan-Welsch, co-principal investigator, NTS 
2005-06 and principal investigator, 2006 -current; Dr. James Rowley, co-principal investigator of the NTS 
2004-05; Dr. Patricia Hart, co-principal investigator of the NTS 2004—06; the principal investigator and co- 
principal investigator of the ExTS, Dr. Sandra Stroot and Dr. Diana Erchick; Dr. Sonja Smith, TQP project 
director 2004-07, Dr. Martha Hendricks, current TQP project director, and Dr. Robert Yinger, TQP research 
director. 
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represent the VAM component of the research. The TQP core research questions outlined in 
TQP documents are as follows: 

1. How do variables of teacher background, initial preparation, and ongoing 
professional learning relate to teacher practices, student learning, and 
achievement? 

2. How do specific elements of teacher preparation and aspects of school contexts 
impact novice teachers’ development during their first 3 years of teaching? 

3. Do HVATs have characteristics, instructional practices, and understandings that 
differ from those of other teachers along the value-added continuum? 

4. What specific school contexts are associated with high value-added novice and 
experienced teachers? 

The five research strands of TQP originally included (a) Graduate Survey Study, in 
which data are collected from preservice candidates and in subsequent years, from inservice 
teachers, about their perceptions of their preservice preparation programs and about their 
experiences as novice teachers in their school settings; (b) Alternative Licensure Study, 
which examines the alternative licensure pathway in Ohio; (c) NTS, which examines the 
practices, characteristics, and professional learning contexts of novice teachers as related to 
P-12 student learning; (d) Experienced Teacher Study (ExTS), which examines the practices, 
characteristics, and professional learning contexts of high value-adding experienced teachers; 
and (e) Large-Scale Longitudinal Study (formerly the Structural Equation Modeling Study), 
which examines the interaction between and among variables including teacher preparation 
program characteristics, instructional practices, school context, and P-12 student learning. As 
of fall 2007, the Alternative Licensure Study strand was completed. 

Examining the effects of teacher preparation in light of impact on P-12 student 
learning is complex. Each study has incorporated a research design, including questions and 
supporting methodology. The data collection measures representing the constellation of 
variables for TQP are found in Figure 1. Note how the key variables of teacher preparation 
(teacher education program features), teacher characteristics (teacher candidate learning), and 
instructional practices relate to pupil learning. Note also the data sources connected with each 
variable. How the NTS contributes to the chain of evidence and policy recommendations is 
described in the next section. 



THE NOVICE TEACHER STUDY 

The purpose of the NTS strand of the TQP research is to examine the instructional 
practices, knowledge and dispositions, and school contexts of HVATs in Ohio schools. A 
second purpose is to examine the professional learning experiences that have contributed to 
this effectiveness. 

In the first section, we describe the design of the NTS and how that design evolved 
from 2004 to 2007. At the end of the first section, we present the research questions guiding 



2 




NTS technical report NTS 07-01 



the NTS. The second section includes a detailed description of the methodology, including 
initial direction for the NTS (2004-05), redirection for the pilot year (2005-06), and 
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Figure 1. Teacher Quality Partnership Evidence Chain. 

changes made in year 1 (2006-07) based on the pilot. Year 2 (2007-08) followed the same 
methodology as year 1. The third section briefly comments on issues and challenges related 
to the methodology. 



Design of the NTS 2004-08 

The NTS research began in 2004. The initial purpose of the NTS was to create 
profiles of practice of beginning teachers. NTS principal investigators pursued development 
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of a field studies code book extrapolated from repeated behaviors documented in extensive 
field notes by a team of researchers. While observing randomly selected novice and 
experienced teachers, the researchers attempted to capture everything they saw and heard in 
these diverse classrooms. From those qualitative notes, principal investigators identified 
categories of teacher behaviors, student behaviors, classroom environments, and available 
resources. The Pis codified the data into a code book on which the researchers were then 
trained, though anything seen or heard that was not in the code book was also to be recorded 
for potential additions to the code book. Using qualitative research methodology, the themes 
and supporting data were to emerge from the evidence. 

A shortcoming of these efforts was that it lacked a connection to other studies of 
effective teachers and entailed the time-intensive task of collecting data, revising the code 
book, training the researchers, and repeating the cycle until little or no new codes were 
needed to reflect classroom evidence. This process required substantial resources to 
compensate researchers, teachers, and support staff. These issues were resolved when TQP 
leadership became aware of a series of research studies being conducted by Pianta (2004) and 
colleagues using a theoretically and empirically grounded observation instrument, the 
Classroom Assessment Scoring System (CLASS). 

In 2005, the NTS aligned more closely with the Structural Equation Modeling Study 
(now the Large-Scale Longitudinal Study); indeed, the NTS participants are considered a 
subset of the LSLS, in that the relationships between and among variables that are being 
established for the LSLS are also being established for the NTS. What distinguishes the NTS 
from the LSLS is that qualitative data are being collected to provide descriptive detail of the 
characteristics, instructional practices, and professional learning contexts of novice teachers. 
As stated in the proposal distributed to external funders: 

The Novice Teacher Study will follow a cohort of approximately 50 new 
teachers for 3 years in order to assess the various contributions of pre-service 
education, induction and mentoring, and school climate and leadership to 
teaching performance and student learning. Attention will also be paid to 
pathways and contributors to professional development and learning. This 
study will seek to develop rich qualitative and quantitative descriptions of the 
specific characteristics of novices’ teaching contexts and practices so that 
once these teachers’ value-added status is determined we can better 
understand the contributions to high student growth in novice teachers’ 
classrooms. Data collection for this study begins September 2005. (Yinger, 

2005, p. 6) 

The following research questions were articulated to guide the NTS: 

1. What instructional practices, knowledge and dispositions, and school 
contexts are characteristic of HVATs in Ohio schools? 

2. What professional learning experiences have contributed to this 
effectiveness? 

The specific NTS research questions are: 
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1. Do teachers who prove to be HVATs have characteristics different from those of 
other teachers along the value-added continuum (e.g., identity as teacher, 
dispositions, vision of teaching)? 

2. Do HVATs have instructional practices that differ significantly from those of 
other teachers along the value-added continuum? 

3. Do HVATs have different understandings regarding the following: 

a. Curriculum? 

b. Subject matter content? 

c. Assessment? 

d. Student diversity? 

e. Instructional contexts? 

f. Differentiation? 

4. Do HVATs have a different orientation to teacher-student relationships (e.g., 
classroom climate, emotional support)? 

5. What specific dispositions and conceptual tools are associated with the 
professional learning of HVATs? 

6. What specific forms of professional development tend to be associated with high 
value-added teaching? 

7. What particular school contexts tend to be associated with the professional 
learning of HVATs? 

In the next section, we describe the methodology of the NTS as it evolved from 2005 
to 2008 to address the NTS research questions. 

Methodology of the NTS 

In this section, we describe the methodology of the NTS from 2005 to 2008. Sections 
include the participants, data collection, data analysis, and dissemination. 

Participants 

NTS participants in the 2005-06 pilot year were third-grade and eighth-grade 
mathematics teachers who volunteered from Ohio school districts that were members of the 
SOAR (Schools’ Online Assessment Reports) consortium. The SOAR School Improvement 
Collaborative is a Battelle for Kids initiative in which a variation of a private enterprise 
(SAS® EVAAS®) value-added analysis is applied to student performance data provided by 
participating school districts. The centerpiece of the project is a secure Web-based database 
that school districts can use to view district-, building-, grade-, and student-level performance 
data. Having access to student performance data was essential since establishing a 
relationship among teacher preparation, teacher variables, and student performance is the 
cornerstone of both the NTS and LSLS. In 2005-06, the pilot year, the NTS recruited 3 
novice teacher participants, and the ExTS recruited 1 1 teacher participants. The primary 
purpose of the pilot year was to refine data collection, and the changes are described in the 
methodology section. 

Four factors resulted in significant changes in the recruitment plan for the NTS and 
LSLS strands for year 1, 2006-07. First, SOAR districts contacted were typically eager to 
involve their novice teachers in TQP, but they rarely had new hires in grades and subjects we 
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had identified for the research. Secondly, SOAR districts where novice teachers were 
available tended to be high-performing districts. While that did not necessarily mean all 
novice teachers would prove to be or to become HVATs, it might have been more likely. 

This could bias the participants toward higher VAM scores, which could impact TQP results. 
Third, the Ohio Department of Education agreed to run a report for TQP from a database it 
maintains with all teacher assignment data. Using this report, the NTS researchers narrowed 
the search for eligible novice teachers to the districts where they had been hired in grade 
levels and subject areas needed for TQP research. Finally, TQP was supporting the SOAR 
recruiting efforts at a high level, whereas the ODE database was made available without cost 
to the project. 

Identifying a pool of novice teachers for the study was not the TQP team’s only 
challenge prior to year 1 of the study. In the pilot year, we had identified teacher participants 
from grades 3 and 8 in reading and mathematics. This decision was made based on student 
performance data that were available to calculate value-added measures at the classroom 
level. The State of Ohio, however, changed its statewide assessment and accountability 
system to be in alignment with federal requirements under No Child Left Behind legislation. 
The stateside assessment system for reading and mathematics spans grades 3 through 8, and 
therefore value-added designations would not be available for teachers in grade 3. The result 
of this decision was, instead of recruiting from grades 3 and 8, we recruited teachers of 
reading, grades 4 through 6, and mathematics, grades 4 through 8. 

School district superintendents have supported our efforts to recruit by granting 
access to the human resources offices that in turn supply names of novice teachers. Building 
principals have also supported our efforts either by supplying names of novice teachers or by 
relaying the invitation to novice teachers in their buildings. This process is labor-intensive 
and is confounded by the fact that entry-year teacher designation in a district may mean that a 
teacher is new to the district but is not a novice teacher. 

In year 1 (2006-07), we recruited 21 novice teachers and invited them to participate 
for 3 years. In November of year 2 (2007-08), we had 23 teachers, 15 of whom returned 
from year 1. The original research plan called for 50 novice teachers, but due to budget 
constraints, we recruited only 23. 

Once a potential participant is identified, he or she is contacted by the NTS project 
office with an invitation to participate. NTS field researchers follow up with a building visit, 
and if the teacher agrees to participate, he or she is given informed consent information 
outlining the study and measures to ensure confidentiality and human subjects’ protection. 
Before data collection begins, teachers are asked to sign two consent forms, one for the NTS 
(Appendix A) and one for the LSLS (Appendix B). 

Data Collection and Analysis 

In this section, we briefly describe the data collection from the pilot year and how we 
refined the data collection based on what we learned from the pilot year. Next we turn to the 
data collection system for the NTS for Year 1, and include a chart in which the research 
questions are aligned with current data collection. We then provide an overview of the data 
collection across an entire year by each of the six data collection cycles, followed by a 
description of each instrument. We include a detailed description of the processes of data 
management, and conclude with a description of data analysis. 



6 




NTS technical report NTS 07-01 



Data Collection Pilot Year (2005-06) 

The instrumentation during the pilot year included: 

1. TQP Preservice Teacher Survey and TQP Inservice Teacher Survey (the 

instruments used in the Graduate Survey Study) 

2. School Physical Environment Checklist 

3. Classroom Physical Environment Checklist 

4. Observation of instruction using the CLASS observation instrument (Pianta, La 

Paro, & Hamre, 2006a, 2006b) 

5. Pre- and post-observation interviews 

6. Teacher and student work samples 

7. Classroom time analysis 

8. Photographs of the classroom environment. 

Data collection measures we intended to collect, but did not, included pedagogical 
content knowledge of reading and mathematics; professional learning; and teacher working 
conditions. 

After the pilot year, the TQP field studies research team reviewed the data from the 
1 1 experienced teachers and 3 novice teachers and made decisions about the data collection 
for year 1 based on importance of data collection to research question; budget; and practical 
issues related to time and resources required for analysis. The following data sources were 
eliminated for year 1 of the study: 

1. Classroom Physical Environment Checklist 

2. Classroom time analysis 

3. Classroom photographs 

4. Teacher and student work samples 

5. Pre-observation interview 

Plans to develop a teacher working conditions survey and the professional learning 
survey were halted. The Content Knowledge for Mathematics Teaching Survey was 
developed for use in year 1. 

Data collection years 1 and 2 (2006-08) 

Data sources for years 1 and 2 of the NTS (2006-08) were: 

1. School Physical Environment Checklist 

2. CLASS observation 

3. Post-observation interview 

4. TQP Preservice Teacher Survey 

5. TQP Inservice Teacher Survey 

6. Content Knowledge for Mathematics Teaching Survey 

The alignment of research questions and data sources is displayed in Figure 2. Each 
data source is described in detail in this section. 

Data collection in the NTS research year is divided into six cycles, or visits. Each 
cycle is displayed in Appendix C, incorporating all data collection occurring in the respective 
cycle. Cycle 1 occurs at the beginning of the school year; it sets the stage for the study in 
which the field researcher establishes the rapport with the teacher participant. The researcher 
also takes note of the physical environment of the school. Cycles 2 through 5 are scheduled 
from early October to April. During these visits, the field researcher observes the teacher 
according to CLASS protocol and records a post-observation interview. Cycle 6 occurs 
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sometime in the last two weeks school is in session. Cycle 6 does not include an observation, 
but the field researcher conducts an unrecorded exit interview. The TQP Preservice Teacher 
Survey and TQP Inservice Teacher Survey and the Content Knowledge for Mathematics 
Teaching Survey are completed in designated cycles. 



Research Questions 


Data Sources 


Teacher characteristics 


TQP Preservice Teacher Survey 
TQP Inservice Teacher Survey 


Teacher practices 


CLASS observations 
Post-observation interviews 


Teacher understandings 


Post-observation interviews 
Content Knowledge for 
Mathematics Teaching Survey 
TQP Inservice Teacher Survey 


Teacher-student relationships 


CLASS observations 
Post-observation interviews 


Teacher professional learning 


Post-observation interviews 
TQP Preservice Teacher Survey 
TQP Inservice Teacher Survey 


Professional development 


TQP Inservice Teacher Survey 
Post-observation interviews 


School context 


TQP Inservice Teacher Survey 
School Physical Environment Checklist 



Figure 2. Alignment of research questions and data collection, NTS years 1 and 2 (2006-08) 



Description of Data Sources 

Teacher profile. During the cycle 1 visit, each teacher, including those novice 
teachers returning to the study for years 2 and 3, completes the teacher profile (Appendix D). 
Profile information consists of descriptive data used to record teacher contact information, 
verify participation eligibility, indicate completion of the TQP Preservice Teacher Survey, 
and provide educational and licensure information. The teacher profile also records 
information needed to process teacher stipend payment at completion of cycle 6. 

School Physical Environment Checklist. The school physical environment checklist 
(Appendix E), completed by the researcher during cycle 1, documents the experience of a 
new visitor to a participating teacher’s school. Adapted from the Fairfax (Virginia) County 
Public Schools’ Parenting Education Center (PEC) Welcoming Atmosphere Walk-Through 
Checklist, the checklist is divided into three parts: the physical environment; the welcoming 
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school staff; and written material. The researcher also includes a summary of her impression 
of the school. 

TQP Preserx’ice Teacher Survey. The TQP Preservice Teacher Survey is administered 
to teacher candidates at the end of their teacher preparation program as part of the Graduate 
Survey Study strand of TQP. The Graduate Survey Study strand personnel check their 
database to confirm that a novice teacher participant completed the survey. If no data are 
located for the participant, he or she completes the survey at the beginning of the year. 

The instrument includes the following: 

• Subscales pertaining to perceptions of the teacher preparation program: (a) coherence 
within program, (b) program quality, (c) field experiences, (d) faculty characteristics, 
and (e) cooperating teacher characteristics. 

• Subscales pertaining to professional knowledge and skills include: (a) motivation, (b) 
curriculum, (c) special education, (d) diversity, (e) literacy, (f) mathematics, (g) 
assessment. 

• Subscales pertaining to teacher efficacy include: (a) classroom management, (b) 
student engagement, (c) instructional strategies, (d) general teacher efficacy, and (e) 
personal teacher efficacy. 

• Subscales pertaining to concerns about teaching include: (a) self concerns, (b) task 
concerns, and (c) impact concerns. 

TQP Inservice Teacher Surx’ey. The TQP Inservice Teacher Survey is administered to 
novice teachers in Ohio in their first year of teaching and in subsequent years according to 
the Graduate Survey Study timeline. Graduate Survey Study participants may complete the 
survey online, but the NTS team decided to ask participants to complete paper copies to 
ensure complete data collection. The survey is distributed in cycle 5 and collected in cycle 6. 
The instrument includes the following: 

• Subscales pertaining to perceptions of the teacher preparation program: coherence 
within program, and program quality. 

• Subscales pertaining to professional knowledge and skills include: (a) motivation, (b) 
curriculum, (c) special education, (d) diversity, (e) literacy, (f) mathematics, and (g) 
assessment. 

• Subscales pertaining to teacher efficacy include: (a) classroom management, (b) 
student engagement, (c) instructional strategies, (d) general teacher efficacy, and (e) 
personal teacher efficacy. 

• Subscales pertaining to instructional orientation-literacy and instructional 
orientation-mathematics include: (a) orientation, (b) activities, (c) materials, and (d) 
knowledge. 

• Subscales pertaining to teaching concerns include: (a) self concerns, (b) task 
concerns, and (c) impact concerns. 

• Subscales pertaining to school context include: (a) collective efficacy, (b) collegial 
leadership, (c) institutional vulnerability, (d) achievement press, (e) teacher behavior, 
(f) trust in principal, (g) trust in colleagues, and (h) trust in clients. 

• Subscales pertaining to working conditions include school characteristics and school 
materials. 
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• Subscales pertaining to professional development include: (a) professional quality, (b) 

professional development impact, and (c) quality of mentoring. 

Classroom Assessment Scoring System ( CLASS). The Classroom Assessment Scoring 
System (CLASS ) is an observational instrument developed to assess classroom quality for 
research and professional development purposes. The CLASS was developed from scales 
used in large-scale classroom observation studies (NICHD Early Child Care Research 
Network, 2002; Pianta, La Paro, & Hamre, 2006a, 2006b), and is “based on developmental 
theory and research suggesting that interactions between students and adults are the primary 
mechanism of student development and learning” (Pianta, et al. 2006a, p. 1). 

According to the developers of the CLASS, interactions between teachers and 
students can be grouped into four domains: emotional support, classroom organization, 
instructional support, and student outcomes. Within each of these domains are specific 
dimensions measured by the CLASS. 

Originally created for use in preschool through the third grade, the CLASS has been 
validated in over 3,000 classrooms from pre-kindergarten through fifth grade. It has also 
been adapted for middle/secondary school and is currently undergoing a similar validation 
process. The use of CLASS across different levels creates, according to Pianta et al. (2006a), 
a “common metric and language for discussion, thereby addressing the problems with grade- 
to-grade transition and the need for coherence” (p. 2). Given that novice teachers participate 
for 3 years, the CLASS approach is particularly relevant because it presents a standardized 
method of organizing classroom observations that tracks teacher performance over time 
(Theroux, 2007). 

Figure 3 displays the CLASS domains and dimensions to reflect the ways in which 
the domains manifest themselves in both elementary and secondary classrooms (Pianta et al., 
2006a, 2006b). Dimensions with no designation are common to both elementary and 
middle/secondary versions. 

The CLASS observation record — elementary and secondary (Appendices F and G) is 
used during cycles 2 through 5. The elementary form is used when observing reading, grades 
4 through 6, and mathematics, grades 4 and 5. The secondary form is used when observing 
mathematics, grades 6 through 8. Each researcher is initially trained according to CFASS 
observation protocols by a qualified CFASS trainer and must score within 80% of agreement 
with a reliability training protocol to qualify as a researcher. Each year of the project, we 
conduct reliability sessions, and again, researchers must score within 80% of agreement on 
the reliability protocol. 

During a CFASS observation, the NTS field researcher observes 20-minute segments 
of a lesson, making note of evidence related to each dimension. The next 10 minutes are 
spent assigning a score of 1 to 7 to each dimension. A low rating is indicated by a score of 1 
or 2; a mid rating is indicated by a score of 3, 4, or 5; and a high rating is indicated by a score 
of 6 or 7. The researcher repeats this 20/10 cycle three to four times in one visit. At the end 
of a research year, we have obtained 12 to 16 scores for each dimension for each novice 
teacher participant. 

Post-obser\’ation inten’iew. The post-observation interview (Appendix H) is 
conducted after the CFASS observation in cycles 2 through 5 and follows the same protocol 
for each cycle. The interview is a semi- structured recorded interview that prompts the teacher 
to consider a lesson taught during a CFASS observation segment. Specifically, the teacher is 
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asked about his or her thoughts on the lesson; the student response(s) to the lesson; 
preparation for the lesson; and decisions made during the lesson, including use of resources. 
The questions are consistent across all interviews, but the researcher may follow a participant 
response with secondary questions and probes, seeking elaboration when necessary. 




Figure 3. Overview of CLASS Domains and Dimensions. 



Content Knowledge for Mathematics Teaching Survey. The Content Knowledge for 
Mathematics Teaching Survey, used by permission of the School of Education, University of 
Michigan, comprises items developed by the UM Study for Instructional Improvement, 
Learning Mathematics for Teaching (LMT), and the Consortium for Policy Research in 
Education. LMT staff provided training for the use of their instruments. 

Two mathematics educators involved in the TQP project 2 compiled the survey 
instrument. This instrument was intended for use by both of the field studies, the NTS and 



2 Diana Erchick and Janet Herrelko 
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the ExTS, and by teachers across multiple grade levels from multiple preparations in terms of 
licensure and certification. To meet the needs of the project, the instrument developers chose 
not to use the full LMT instruments made available by the University of Michigan group and 
instead chose to compile an instrument to specifically meet the needs of the TQP studies. 

This decision was based on the following needs: 

1. Sensitivity to teacher time constraints, both in terms of their roles as classroom 
teachers and as participants in our study, required an instrument that could be 
completed by the teacher in about 60 minutes. The TQP instrument is composed 
of 20 questions, each expected to take about 3 minutes to complete. 

2. The TQP project needed an instrument that included mathematics content across 
as much of the curriculum as possible. The LMT instruments focused on 
particular content strands; thus, we would have had to administer multiple surveys 
to be inclusive of the content. 

3. The TQP instrument would be administered to teachers whose teaching 
experience covered a wide range of possibilities. The LMT instruments were 
designed for groups of teachers at either the elementary certification or middle 
school licensure levels. 

4. Each of the groups of TQP participating teachers (novice and experienced) comes 
from a different teacher preparation background. The novice teachers all have 
licenses that differ significantly from the experienced teachers’ certifications, 
particularly in mathematics content preparation. The preparations also differ 
within each of the licensure and certification structures, so it was important to 
design the TQP instrument independent of attention to teacher preparation. Thus, 
little attention was focused on the difficulty levels of TQP instrument items. 

In compiling the items for the TQP instrument, the mathematics educators reviewed 
the items in the LMT instruments from the University of Michigan and selected items 
believed to address the five content standards of the National Council of Teachers of 
Mathematics (NCTM). As noted above, the LMT instruments are not designed to address all 
content. However, selected items often address content beyond the intended, and from those 
items, the TQP designers were able to choose items to complete the full range of content 
areas needed by TQP. 

It can be argued that in addition to assessing content knowledge, completion of many 
of the LMT items also requires use of the mathematical process. Although the LMT project 
does not claim to address the NCTM’s mathematics process standards, the TQP mathematics 
educators recognized the role of processes in the work needed to complete the LMT items. 
This recognition was based on the guidance of the Principles and Standards for School 
Mathematics (NCTM, 2000) document, the Ohio Academic Content Standards, and recent 
research. In addition to attending to content needs, the mathematics content survey designers 
attempted to include the full range of process standards in the items chosen for the TQP 
instrument. The only process standard not well-represented in the TQP instrument is the 
communication standard. This is primarily because of the nature of the TQP instrument 
(forced response as opposed to short answer or extended response and written as opposed to 
oral). 
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Field notes. During every cycle, the researcher may complete a separate page of field 
notes. As described in the NTS data collection manual, the field notes give the researcher an 
opportunity to record whatever is noteworthy during any given visit. Researchers may record 
both descriptive notes and reflective notes. A descriptive note is a non-judgmental record of 
an observation. A reflective note is the researcher’s explanation of why she made note of a 
particular observation. 

We are in the process of analyzing the data for year 1, described in the next section. 
We note here that the data collection for year 2 (2007-08) of the NTS follows the same cycle 
and protocol used in year 1. 

Data Analysis 

School Physical Environment Checklist. Items on the School Physical Environment 
Checklist (Yes, No) were entered in a database and frequencies were calculated for each item 
and for the total checklist score. The results are considered in constructing case studies of 
high value-added teachers. 

TQP Preservice Teacher Survey and TQP Inservice Teacher Survey. For those novice 
teacher participants who completed the TQP Preservice Teacher Survey during their 
preservice teacher education program, the Graduate Survey Study research team confirmed 
that they had the data for those participants. Those novice teachers who had not completed 
the survey during their teacher preparation program completed a paper copy of the survey, 
and the NTS team entered the data into a template provided by the Graduate Survey Study 
team and sent it to The Ohio State University for analysis. Analyses included descriptive 
statistics (maximum score, minimum score, average score, and standard deviation) on each 
survey item and the scales. Separate analyses were run for novice teachers completing the 
TQP Preservice Teacher Survey during their preservice preparation program, novice teachers 
completing the survey in cycle 1, and the total group. We followed this same process for the 
TQP Inservice Teacher Survey. 

CLASS. CLASS dimension scores for each participant are aggregated into four 
categories, or domains: emotional support; instructional support; classroom management; and 
student outcomes. For each CLASS domain, descriptive statistics were calculated that 
included the minimum score, maximum score, average score, and standard deviation. 
Descriptive statistics were created for both novice and experienced teachers; separate /-tests 
for equal means were used to determine if the two groups, novice and experienced teachers, 
are significantly different. In addition, correlation matrices were created to study the 
relationship among the four CLASS domains. 

Content Knowledge for Mathematics Teaching Survey. Descriptive statistics of 
participant scores were calculated, including the minimum score, maximum score, average 
score, and standard deviation. Descriptive statistics were created for both novice and 
experienced teachers’ scores, separately. 

A /-test for equal means was run to determine if the scores on the Content Knowledge 
for Mathematics Teaching Survey are different for novice and experienced teachers. Finally, 
for both novice and experienced teachers, the mathematics content knowledge scores were 
included in the correlation matrices of the CLASS observations. 
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Post-obser\’ation interview data. At the release of this technical report, the TQP 
researchers were in the process of analyzing the qualitative data from year 1 using widely 
accepted qualitative data analysis techniques: identifying codes; marking segments of texts 
with codes; and organizing coded data to determine themes or patterns at more abstract levels 
of interpretation (Miles & Huberman, 1994; Ryan & Bernard, 2000). 

Key personnel of the NTS and ExTS defined an initial set of codes after formatting 
data from the year 1 post-observation interview for analysis. Codes were post-defined (not 
specified before or during data collection), a method recommended by many scholars 
(Anfara, Brown, & Mangione, 2002; Patton, 1990). However, we do note that the research 
purpose and theories guiding the development of the project, including the interview 
questions, helped to determine some of the coding categories used before the open coding of 
transcribed interviews began. 

Researchers agreed that the text segments to be coded were a complete turn — the 
entire text of a question posed by the field researcher and the corresponding teacher 
participant response. By coding an entire turn, researchers preserved prompts, contextual 
data, and other information that would help with interpretation. 

Field researchers from the NTS and the ExTS met with key personnel from both 
studies to explore the coding scheme; challenge code use and interpretation; and develop the 
integrity of the codes. Difficulties in establishing an acceptable level of inter-rater reliability 
occurred. While the researchers were typically in agreement concerning the use of some 
codes — resources, standardized test, and instruction, for example — agreement about 
consistent use of other codes remained elusive. After much discussion, researchers were able 
to attain a rate of 90% reliability using the Miles and Huberman (1994) equation, agreeing 
upon a system that used codes as broadly and descriptively as possible. This solution was 
acceptable because of the use of the qualitative analysis software NVivo. NVivo allows 
coding at multiple levels, and the 15 codes agreed upon constituted the “parent” codes. The 
subsequent use of “child” codes assures the research team that the more subtle distinctions in 
the data would not be lost. In alphabetical order, the 15 codes are: (a) Administration, (b) 
Classroom Context, (c) Content/Subject Matter, (d) Curriculum, (e) Instruction, (f) No Code, 
(g) Parents, (h) Planning, (i) Professional Learning, (j) Resources, (k) School Context, (1) 
Standardized Testing, (m) Student Characteristics, (n) Student Response, (o) Teacher 
Characteristics. See Appendix I for code book with rules and exemplars. 

Field notes. Field notes are being analyzed using the same process developed for the 
interviews. 



Data Management 

In this section, we describe the NTS data collection and management process 
developed in the pilot year 2005-06, refined in year 1 (2006-07), and again refined and 
implemented in year 2 (2007-08). Each year provided new insights into the needs of the 
study and how to manage the data to assure complete and high-quality data. 

The data collection process involved creating a physical and electronic system at 
the University of Dayton, Dayton, Ohio. The NTS system development included the 
combined efforts of the key personnel for the TQP field studies (see footnote 1), NTS field 
researchers, and the data management team. The systems created included the following: 
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1. Data collection protocols and tools linked to the research questions and desired 
outcomes 

2. Procedures for using the tools to collect data 

3. Regular communications with field researchers 

4. Data retrieval processes to ensure complete data sets across the subjects 

5. Data storage designed for accessibility, security, and analysis 

6. A system of communication to facilitate using the data for analysis and to ensure 
data integrity 

Data were collected by a group of NTS field researchers. To guide the process of data 
collection, the TQP team developed data collection protocols, or procedures, that 
accompanied each data source described in a previous section. We reviewed these procedures 
at monthly meetings of the TQP Pis and during regular training sessions with the field 
researchers. Through regular e-mail communications we clarified procedures and other 
matters as questions arose. 

The NTS team compiled a data collection manual, which contained all protocols and 
instruments arranged by cycle. The manual provided explicit directions for the field 
researchers, thus ensuring consistent data collection across all teacher participants. During 
2006-07, the NTS research group 3 began meeting regularly at the University of Dayton to 
discuss data collection progress, review teacher interview data samples, consider suggestions 
to streamline data collection, and discuss issues relevant to the data collection process. Out of 
these meetings came significant refinements to the data management process, including 
creating a framework for labeling the CLASS observation segments, establishing guidelines 
to enhance the quality of field notes and interview data, and creating a consistent file 
identification system. The meetings further built the trust and camaraderie of the study group 
and allowed for discussion of more sensitive issues such as the researcher-participant 
relationship and requests by the teacher participants for feedback on the observed instruction. 
The data collection experiences of NTS year 1(2005-06 led to several refinements in NTS 
year 2 (2007-08). 

For each data collection cycle, the field researcher is provided a cycle checklist, 
indicating the data to be collected that cycle and the specific tasks to be accomplished. The 
checklist is a tool for the field researchers to determine that the appropriate data are collected, 
surveys are distributed for collection in the next cycle, or upcoming visits are scheduled. 

Each data source is submitted on a form that supports systematic recording and retrieval of 
data for analysis. Researchers are also provided electronic and hard copies of all forms. 

The NTS project office created systems for filing electronic records and hard copies 
to ensure secure storage and retrieval of the data. The electronic records are maintained on a 
separate password-protected computer drive accessible to the PI and the NTS project office 
staff. The drive is backed up weekly using an external hard drive. 

The electronic record-keeping system is composed of a teacher database; individual 
teacher folders containing a separate folder for each cycle and all submitted data by cycle; an 
individual teacher data-tracking sheet; and a comprehensive data-tracking sheet containing 
confirmation on data collected for each teacher by cycle. A systematic process for recording 
submitted data includes verifying the data by matching teacher ID number, field researcher 



1 K. Kinnucan-Welsch, PI; field researchers: P. Ellis, D. Frank, K. Pareso, B. Shervey; data manager: C. Currell. 
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ID number, data and form for submission, cycle number, and data collected. If there were 
any issues pertaining to the data, such as missing ID number, incomplete data, etc., the data 
manager communicated with the field researcher and resolved it. Data were then filed, hard 
copy and or electronically. The NTS project staff conducted two formal data audits and 
ongoing informal data audits to ensure that the appropriate data were submitted, received, 
and recorded. Personal identifying data such as teacher name, school, individual students, 
principal, or other educators in the building are omitted from all NTS data. 

In order to provide a consistent reference for all data, the following naming logic is 
used: Teacher ID number_cycle number_form number_date data were collected. 

This logic was particularly important with the CLASS observations and the post- 
observation interview. Researchers identified CLASS segments in relation to the groups of 
students being observed. For example, each 20-minute CLASS observation segment ID 
number includes the teacher ID number, the subject observed (reading or math), and an 
indication of when the group of students changes. A CLASS observation with the file name 
NT07-015_cy2_form 7b_9-25.07_Ml signifies the following: Data collected are from 
returning novice teacher number 015, who began participating in the study in the 2006-07 
school year. The data collection cycle is cycle 2. The data collected are the CLASS notes 
observation data for secondary students. The CLASS observation took place on September 
25, 2007. This file contains the observation data for math class segment Ml. 

Since the pilot year in 2005, the NTS group has used a process of shared learning and 
experience from the field to guide the development of the systems currently in place. The 
current system is a result of ongoing analysis of data quality, and the entire NTS research 
team has contributed to the improvements in the system from the pilot year to date. 

Dissemination 

The progress of the NTS, including refinement of the research questions and 
methodology, has been presented annually to the TQP external audit panel and semi-annually 
to the Ohio advisory board. Preliminary findings from the pilot year data were presented at 
several regional and national conferences, including the American Association of Colleges of 
Teacher Education (AACTE); Battelle for Kids Power of Two; Ohio Confederation of 
Teacher Education Organizations; REL Midwest board in Chicago; Kentucky Association of 
Colleges for Teacher Education; New Teacher Conference sponsored by the Ohio 
Department of Education with funding from the Joyce Foundation; and the Holmes 
Conference. 

A series of technical reports is scheduled for release in 2007-08, beginning with this 
report summarizing the background and research. Publications drawn from the technical 
reports will be submitted to peer-reviewed journals in 2008. 



REFLECTIONS ON THE METHODOLOGICAL ISSUES AND CHALLENGES 

We have encountered several challenges in the NTS research. Many of these 
challenges are being noted by others attempting to examine teacher quality from a complex 
perspective [see themed issues Evidence, Efficacy, and Effectiveness, Journal of Teacher 
Education , 57(1), 57(2)], We discuss the following in our concluding remarks: recruiting; 
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quality of data; and building converging evidence through use of common instruments for 
data collection. 



Recruiting 

As noted in a previous section of this report, recruiting novice teachers has posed 
many challenges. Many stakeholders are supportive and involved in the process, including 
TQP institutional representatives and field office directors at Ohio institutions of higher 
education; Ohio Department of Education personnel; and regional and local school district 
administrators and human resources personnel. One dilemma is that schools are not required 
to submit lists of entry year teachers to the Ohio Department of Education until well into the 
school year, thus eliminating what would otherwise be a comprehensive list from which to 
locate teachers. The issue here is not that we don’t have the list; it is that it is important to 
collect data on the novice teachers at the beginning of the school year, and the list is not 
available at that time. 

Local district administrators and human resources personnel have been essential to 
our recruiting efforts. We have been most successful in those districts where the lead 
universities in the project (the University of Dayton, the University of Cincinnati, Wright 
State University, The Ohio State University, and Mount Vernon Nazarene University) have 
established relationships with the administrators. Field studies such as the NTS require 
researcher access, and personal contact is certainly one avenue to gain that access. The 
negative impact is that participants tend to be recruited from districts strongly affiliated with 
the universities, which could have an influence on the school context in which the novice 
teachers are working. Actions we are taking to support our recruiting include more exposure 
of the research project to local districts and tighter connections with the Ohio Department of 
Education. 



Quality of Data 

The NTS is one strand of a complex, large-scale study. The NTS study is a subset of 
the Large-Scale Longitudinal Study, in which teacher preparation, teacher characteristics, 
and instructional practices are linked to teacher quality through structural equation modeling. 
It is essential that we follow data collection, storage, and retrieval processes that assure high- 
quality data for both the NTS and the LSLS while maintaining confidentiality of participants 
per institutional review board assurances. The data management processes necessary to meet 
these requirements are resource-intensive — an issue that must be considered in allocating 
funds for the research. 

The training requirements and ongoing contact with the field researchers are also 
worth noting. We decided early in the study that it was important to have the researchers in 
close proximity to the University of Dayton so that regular contact with the principal 
investigator was feasible. This choice confirmed the decision to recruit in an area close to the 
University of Dayton, and as mentioned in the section on recruiting, this limited the 
geographic region from which participants are drawn. 
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Creating Opportunities for Converging Evidence 

The TQP research team made a decision to use two instruments that had been 
developed and used in previous research: the CLASS and the Content Knowledge for 
Mathematics Teaching Survey. The Graduate Survey Study also incorporated previous 
instruments into the TQP Preservice Teacher Survey and TQP Inservice Teacher Survey. By 
building on previous work, educational researchers, when appropriate, will be able to connect 
research findings across studies. This will support a deepening knowledge base on teaching 
and teacher education, a need highlighted by educational researchers, policymakers, and the 
public. 

To conclude, the information contained in this technical report provides details about 
the design and methodology of the NTS. It is our intent that stakeholders will find this 
information useful when reviewing TQP research reports and considering policy implications 
that emerge from the TQP research. 
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APPENDIX A 

Teacher Quality Partnership: 
Novice Teacher Study 
Novice Teacher Consent Form 2007-2008 



Dear teacher participant: 

Thank you for agreeing to participate in the Novice Teacher Study. The purpose of this study is to 
understand the practice of beginning teachers, and the factors influencing that practice. We recognize 
the challenges facing entry-year teachers, and we have structured data collection to minimize 
intrusion as much as possible. A complete list of the data collection cycles is attached to this consent 
form. If you agree to participate, please read the information below, sign, and return to your field 

researcher. Thank you for contributing to the profession of teaching. 

Name: School: 

TQP Agreement: 

1. Maintain complete confidentiality. All reports will be summary data only. At no time will 
any teacher or school be identified. 

2. Arrange the interviews and observation at a time and days that are convenient to the 
teacher. 

3. Pay teacher participants $300 (year 1), $500 (year 2), $700 (year 3) when the agreement 

has been completed. 

Teacher Agreement: 

I commit to collect data for the Novice Teacher Study. I understand that I: 

■ Must participate in 6 data collection cycles between September, 2007 and June, 
2008. These cycles include observations of teaching with the CLASS 
observation protocol and post-observation interviews following the observations. 

■ Complete all surveys that have been approved for use in the study, including the 
TQP Inservice Teacher Survey. 

■ I recognize that the data collected in the Novice Teacher Study will be used for 
the purposes of the research only. When the analyses of the data are conducted, 
the identity of the participants will remain confidential to the Principal 
Investigators and field researchers, and will not be revealed in any reports of 
research or to anyone outside of the study. 

■ Participation is voluntary, refusal to participate will involve no penalty or loss of 
benefits to which I am otherwise entitled, and I may discontinue participation at 
any time without penalty or loss of benefits to which I am otherwise entitled. 

■ I should contact the Principal Investigator, Dr. Kathryn Kinnucan-Welsch, at 
Katie. Kinnucan- W elsch @ notes.udayton.edu for answers to questions about the 
research. 

■ I should contact for questions about my rights as a subject or in the event of a 
research-related injury to the subject, Mr. Jon Nieberding, University of Dayton 

Institutional Review Board Chair, at (937) 229-4053. 



I, 



(Print Name) 



agree to participate in the Novice Teacher Study. 



Teacher’s Signature 



Date 



Principal Investigator Signature 



Date 
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APPENDIX B 



Consent to Participate in a Research Study - Novice Teachers 

University of Cincinnati 

College of Education / Department of Educational Leadership 
Suzanne Franco, Ed.D. 

937-775-3673 (Suzanne.franco@wright.edu) 



Title of Study: TQP Large-scale Longitudinal Study of Novice Teachers 

Purpose of Study: To better understand the aspects of teacher preparation and early career 

support that help new teachers be successful in teaching math and/or reading 
to elementary and middle school children. 

Participants: Any new teacher of 4 th , 5 th , 6 th , 7 th , or 8 th grade mathematics in an Ohio school 

who has just graduated from an Ohio teacher preparation program (not an 
alternative license program). 

Any new teacher of 4 th , 5 th , or 6 th grade reading in an Ohio school who has just 
graduated from an Ohio teacher preparation program (not an alternative 
license program). 

By signing below, I am agreeing to participate in the Teacher Quality Partnership “Large-scale 
Longitudinal Study” of new teacher preparation and support in Ohio. I have read and understand the 
Research Information Sheet - Teachers provided by Dr. Ted Zigler, researcher, on behalf of Dr. 
Suzanne Franco, the leader of the study. 

I understand that everything that I do with this study will be kept strictly confidential, that all 
surveys and information about me will be coded to avoid identification, and that all records (including 
this consent form) will be stored in locked cabinets and facilities at the University of Cincinnati 
throughout the period of the study, and for two years afterward. 

I understand that I am not obligated to the study in any way, and may leave the study at any 
time, with or without giving a reason. 

I understand that if I have any questions about study-related activities, I can call Dr. Suzanne 
Franco, Ed.D. 937-775-3673 or email her at Suzanne.franco@wriqht.edu . If I have questions about 
my rights as a research participant, I know that I can call the Chair of the Institutional Review Board - 
Social and Behavioral Sciences at (513) 558-5784. 

I understand that my participation in this research study will be for three years (unless I should stop 
teaching in Ohio), and that I will be asked to complete a survey and to confirm my students in each of 
those three years. These activities will only take me about an hour total each year. There are no 
alternative activities; if I cannot complete the regular study activities, I can opt to leave the study. 

Finally, I understand that there are no expected risks or benefits to me personally from 
participating in this study. If I am interested in finding out about the study progress, I know that I can 
check the study website at www.tqpohio.org for regular reports. 



Study Participant / DATE Witness 



/DATE 
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2007-08 


Cycle 1 


Cycle 2 


Cycle 3 


Cycle 4 


Cycle 5 


Cycle 6 


Timeframe 

to 

complete 

data 

collection 


before start of / or early in 
school year 
2007 


mid-Sept to early Oct 
2007 


early to mid- 
Nov 2007 

(>/=10 days prior to 
Thanksgiving) 


Feb-08 


April 2008 
after 

spring break 


within the 
last 2 weeks 
of school 


Focus 

of 

Cycle 


Setting stage for 
data collection 
- initial building visit. 


One (1) day of CLASS 
observation 
and teacher interviews 


One (1) day of CLASS 
observation 
and teacher 
interviews 


One (1) day of 
CLASS observation 
and teacher 
interviews 


One (1) day of 
CLASS observation 
and teacher 
interviews 


Final visit and conclusion 
of study school year 




Initial Teacher Meeting 
and Interview 

la 

Text Entry Form 

form 1b 


CLASS Notes Sheet 

form 7 

CLASS Submission 

form 8 


CLASS Notes Sheet 

form 7 

CLASS Submission 

form 8 


CLASS Notes Sheet 

form 7 

CLASS Submission 

form 8 


CLASS Notes Sheet 

form 7 

CLASS Submission 

form 8 


Final Visit 

protocol 14 

Collect final data. Gain 
teacher sign off on Data 
Checklist. 

form 15 




NTS Informed Consent 

form 2 

LSLS Informed Consent 

(UC form) 


Post-observation 

Interview 

9a 

Data Entry 

form 9b 


Post-observation 

Interview 

9a 

Data Entry 

form 9b 


Post-observation 

Interview 

9a 

Data Entry 

form 9b 


Post-observation Interview 

9a 

Data Entry 

form 9b 


Collect Intent to 
Participate and Summer 
Contact information 

form 16 


Data 

Collection 

Protocols 


School Physical 
Environment Checklist 

form 4 


Field Notes 
Text Entry Form 3b 


Field Notes 
Text Entry Form 3b 


Field Notes 
Text Entry Form 3b 


Field Notes 
Text Entry Form 3b 


Field Notes 
Text Entry Form 3b 


and Tools 


Teacher Profile 

form 5 








Distribute TQP Inservice 
Teacher Survey 

form 12 


Collect TQP Inservice 
Teacher Survey 

form 12 




Verify if teacher 
completed TQP 
Preservice Teacher 
Survey at IHE 

form 5 


Distribute 
TQP Preservice 
Teacher Survey to 
those w/o confirmed 
survey 
protocol 6 


Collect 

TQP Preservice 
Teacher Survey from 
those given in Cycle 2 

protocol 6 


Distribute Content 
Knowledge for 
Mathematics 
Teaching Survey 

form 1 1 


Collect Content Knowledge 
for Mathematics Teaching 
Survey 

form 1 1 






Schedule date 
for C 2 


Schedule date 
for C 3 


Schedule date 
for C 4 


Schedule date 
for C 5 


Schedule date 
for C 6 


FR Submit all data to 
Project Office. PI sign 
off for FR payment 

form 18 
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Teacher Quality 
Partnershi 




NTS Data Collection 2007-08 
Teacher Profile 
Cycle 1 



Last Name: 



School: 



First Name: 



DOB: / / 

SSN: 



District: 

Current teaching assignment by grade level and subject: 



Grade Level: 


Subject: 


Grade Level: 


Subject: 


Grade Level: 


Subject: 



*Note: all identifying information will be 
kept in locked files and will only be used 
for obtaining study data from ODE and to 
make stipend payments. 



Email at school: 

School phone: ( ) School fax : ( ) 



Home mailing address: 



Home Email: 

Phone: cell ( ) home ( ) 

Licensure: 



Early Childhood Education (ECE) 
Middle Childhood Education (MCE) 
Adolescence Young Adult (AYA) 
Intervention Specialist (IS) 

Other: please describe: 



Name of licensure granting institution: 



Date of completion of licensure: / / 
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UNDERGRADUATE EDUCATION : (list all undergraduate education) 

1. Undergraduate: 

Name of Institution dates attended year of graduation 

Degree: Major: Specialty area: 

2. Undergraduate: 

Name of Institution dates attended year of graduation 

Degree: Major: Specialty area: 

3. Undergraduate: 

Name of Institution dates attended year of graduation 



Degree: 



Major: 



Specialty area: 



Degrees 

BS Education 
BA Education 

BS degree if other than education: 

BA degree if other than education: 

Other degree: Area of study: 

Majors 

Early Childhood Education (ECE) 

Middle Childhood Education (MCE) 

Adolescence Young Adult (AYA):specialty area(s): 
Intervention Specialist (IS) 

Other: please describe 



GRADUATE EDUCATION: 



1. Master’s Degree: 

Name of Institution 



dates attended 



Degree: 



Area of study: 



2. Master’s Degree: 

Name of Institution 



dates attended 



Degree: 



Area of study: 
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Weekly Schedule - Sample to assist in scheduling school visits and observation. 
Note: This schedule is for Field Researcher to keep for own reference. 



Time 


Monday 


Tuesday 


Wednesday 


Thursday 


Friday 











































































Contact Notes: 
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APPENDIX E 

School Physical Environment Checklist 

NT08-Cycle 1 

Field Researcher ID #: 

Date of Visit 



Physical Environment 




Yes 


No 


Comments 


Signs giving clear directions to the 
main office are posted near the front 
entrance and at other entrances. 








A welcome sign is displayed near the 
entrance. 








There are clear directions for all 
visitors to sign in at the office and 
obtain a building pass. 








There is a bulletin board on which 
parents and parent organizations post 
and receive news and 
announcements. 








Bulletin boards and displays 
throughout the building are student- 
oriented, colorful, and well- 
maintained. 








There is a place where visitors can 
comfortably sit to chat, read available 
resources, prepare materials for 
teachers, etc. 








The building and grounds are clean 
and in good repair. 








During periods when groups of 
children are in the hallways, there is 
evidence of respect. 








The measures of security are 
appropriate. 








Welcoming School Staff 


The office staff greets visitors quickly 
with a smile and in a friendly, 
courteous, way. 








Staff members passing in the hall 
acknowledge visitors with a smile, a 
nod or a hello. 
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School Physical Environment Checklist 

NT08-Cycle 1 



Parents are visible in the school 
building. 








Written Materials 




Yes 


No 


Comments 


All printed materials are clear and 
understandable to someone who is 
new to the school. 








A variety of school programs are 
highlighted, including special 
education, music programs, general 
education, English as a Second 
Language, etc. 








There is obvious collaboration with 
the school’s PTA and other parent 
groups. 








There is obvious collaboration with 
the community. 








Photographs and articles highlight the 
diversity of the student body. 








Student work is highlighted in the 
publications. 








The school’s educational and extra- 
curricular programs are explained. 








Translated publications are readily 
available and distributed to families 
who have been identified as needing 
them. 








The printed materials use a font that 
is easy to read and the format is neat 
and clean. 








The school’s website is accurate, 
updated and provides useful 
information to families and students. 
(Homework information, school 
calendar, etc.) 
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School Physical Environment Checklist 

NT08-Cycle 1 



Please record and submit responses to following two questions on the field notes 
data entry form 3b. 



1 . Summarize your impression of the atmosphere of this school. 



2. Are there other comments you would like to add? 



This list was modified extensively from the Fairfax (Virginia) County Public 
Schools’ Parenting Education Center (PEC) Welcoming Atmosphere Walk- 
Through Checklist. 
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APPENDIX F 



CLASS Observation Record - Elementary 

CLASS Notes Sheet 
TOP Data Collection NT08 (2007-08) 



Teacher ID #: 



Date of Observation: 



Cycle #: 



Field Researcher ID 
#: 



No. Adults in Room: 



Use for Post-Ob Interview □ 



Segment #: 
File Name#: 



Time of Segment (start/end): 



/ 



Briefly describe what was happening in the classroom during this segment. 



Positive Climate 


Evidence Mark “x” next to desired score 


Score 


• 


Relationships 




1 


• 


Positive affect 




2 


• 


Respect 




3 


• 


Positive peer interactions 




4 

5 

6 
7 


Negative Climate 


Evidence 


Score 


• 


Negative affect 




1 


• 


Punitive control 




2 


• 


Sarcasm/disrespect 




3 


• 


Negativity not connected to 




4 




events 




5 


• 

• 


Negativity escalates 
Severe negativity 




6 

7 


Teacher Sensitivity 


Evidence 


Score 


• 


Responsiveness 




1 


• 


Notices when students need 




2 




assistance 




3 


• 


Appropriate activities 




4 


• 


Addresses problems 




5 


• 


Students seek support 




6 


• 


Student comfort 




7 


Regard for Student Perspectives 


Evidence 


Score 


• 


Flexibility and student focus 




1 


• 


Support of autonomy 




2 


• 


Student expression 




3 


• 


Student responsibility 




4 


• 


Peer interaction encouraged 




5 


• 


Restriction of movement 




6 

7 


Behavior Management 


Evidence 


Score 


• 


Proactive 




1 


• 


Monitoring 




2 

Q 


• 


Redirecting misbehavior 




o 

4 


• 


Clear behavioral expectations 




5 


• 


Loss of time 




6 


• 

• 


Effective praise 
Student misbehavior 




7 
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Productivity 


Evidence 


Score 


• Provision of activities 




1 


• Routines 




2 


• Transitions 




3 


• Preparation 




4 


• Disruptions 




5 


• Managerial tasks 




6 

7 


Instructional Learning Formats 


Evidence 


Score 


• Utilization of materials 




1 


• Student engagement 




2 


• Clarity of learning objectives 




3 


• Teacher facilitation 




4 


• Modalities 




5 






6 






7 


Concept Development 


Evidence 


Score 


• Higher order thinking and 




1 


cognition vs. rote learning 




2 


• Analysis and reasoning 




3 


• Hypothesis testing 




4 


• Integration with previous 




5 


concepts 




6 


• Connections to the real world 




7 


Quality of Feedback 


Evidence 


Score 


• Process of feedback 




1 


• Feedback loops 




2 


• Specific feedback 




3 


• Providing hints 




4 

5 






6 






7 


Language Modeling 


Evidence 


Score 


• Frequent conversation 




1 


• Student-initiated language 




2 


• Open-ended questions 




3 

4 


• Repetition and extension 




5 


• Self and parallel talk 




6 


• Advanced language 




7 


Student Engagement 


Evidence 


Score 


• Active vs. passive engagement 




1 


• Sustained engagement 




2 

q 






U 

4 






5 






6 






7 
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APPENDIX G 

CLASS Observation Record - Secondary 
CLASS Notes Sheet - TQP Data Collection NT08 (2007-08) 



form 7b 



Teacher ID#: 



Date of Observation: 



Cycle #: 



Field Researcher ID#: 



No. Adults in Room: 



Use for Post-Ob Interview □ 



Segment #: 
File Name: 



Time of Segment (start/end):_ 



Briefly describe what was happening in the classroom during this segment: 



Positive Climate 


Evidence Mark “x” next to desired score 


Score 


• Relationships 




1 


• Positive affect 




2 


• Positive peer interactions 




3 


• Interest in students’ lives 




4 

5 

6 
7 


Negative Climate 


Evidence 


Score 


• Negative teacher affect 




1 


• Punitive control 




2 


• Negative student behavior 




3 


toward peers and teacher(s) 




4 


• Teacher response to student 




5 


d iscri m i nato ry/i nf la m mato ry 




6 


behavior 




7 


Teacher Sensitivity 


Evidence 


Score 


• Anticipation of and 




1 


responsiveness to students’ 




2 


academic/social/emotional needs 




3 


and cues 




4 


• Notices when students need 




5 


assistance 




6 


• Effectiveness in addressing 




7 


problems 






• Students’ comfort in seeking 






support and participating 






Regard for Adolescent Perspectives 


Evidence 


Score 


• Opportunities for decision- 




1 


making 




2 


• Relevance- 




3 


Usefulness/connection to current 




4 


life 




5 


• Attention to ST ideas/opinions 




6 


• Meaningful peer interactions 




7 


• Student action options 
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Behavior Management 


Evidence 


Score 


• 


Proactive 




1 


• 


Clear behavioral expectations 




2 


• 


Monitoring 




3 

4 


• 


Redirecting misbehavior 




5 


• 


Loss of time 




6 


• 


Effective praise 




7 


• 


Student misbehavior 
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APPENDIX H 



Post-observation Interview 



Background Notes to Field Researcher 

The purpose of this interview is to gain insight into the teacher’s professional 
practice by inviting him or her to reflect on a teaching episode he or she has just 
completed and you have just observed. Specifically, we are interested in 
understanding teacher practice in the post-teaching context. The interview should 
produce expanded account field notes that provide deep insight into the above. 

> Please audiotape the interview. Save the audio file using the audio file save 
procedures. Submit the Word document from your own transcription of this 
interview according to the electronic data submission procedures. Check to 
make sure you have adequate battery power and good sound levels on 
your recorder before beginning the interview. Periodically check the recorder 
to make sure you are still recording. Also be sure you are recording in a 
digital file that has adequate space to record the interview. 

> As the teacher’s responses unfold, whenever possible, probe as to where he 
or she learned how to do the things he or she Is describing. 

> IMPORTANT: This is a semi-structured interview protocol. As you listen to the 
subject’s responses, be prepared to use secondary prompts like the ones 
listed below to elicit additional explication of the phenomenon of interest. 

> You will focus on one lesson for your interview. Give your teacher the 
opportunity to suggest which lesson he/she would like to talk about. If there is 
no preference, be prepared to select one and state your reason why in your 
field notes. 



Possible Field Researcher introduction: 

I would like to begin with the reminder that the purpose of this interview is not to 
evaluate you, but instead to simply try to better understand teacher practice and 
where you learned about your practice with regard to the observed lesson. Any 
details you can share regarding your role in the lesson and the performance of the 
students would be most helpful. 
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Question #1 : 

Ask the teacher if there is a lesson he/she would like to talk about. If not, identify a 
lesson you would like to talk about. 

Think about the focus of this lesson (specify which lesson) I observed. How do you 
think it went? 

Question #2: 

How did the students respond to this lesson? 

Prompts: 

Did the students learn what you were trying to teach them? 

If they respond “yes,” “no,” or “some of them learned,” probe: How could you tell? 
What makes you think so? Tell me more about that. 

If they respond “I don’t know,” probe again: What makes you unsure? Tell me more 
about what you are thinking here. . . 

Question #3 

What decisions did you make during the course of the lesson? 

Prompts: 

What prompted this (these) decisions? 

Probe for specifics here, there may have been multiple decisions and different 
reasons for each of the decisions made during the lesson. 

Were there any surprises in this lesson? Things you did not anticipate? 

Did you do anything you didn’t plan? 

Question #4 

Talk a little bit about the materials and classroom resources you used in this lesson. 

Prompt: 

Is there anything you did not have that you wish you would have had? 



Question #5 

Where did you learn the most about how to teach this lesson(s)? 

Prompt: 

Probe to see how the teacher learned about the content that was taught and the 
strategies used to teach the lesson. 
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Question #6 (If teacher identified the lesson for interview, then ask this question.) 

Now that we have talked about this lesson, tell me why you selected it. 

General prompts that can be used at any time during the interview process: 

> Tell me a little bit more about that . . . 

> What do you mean by that . . . ? 

> Can you give me an example of what you mean here . . . ? 

> Can you expand on what you mean by that comment . . . ? 



Reminder: We want to utilize the interview to better understand the participants’ 
views on the various topics. We want to learn from them. The researcher’s job is to 
probe without leading the response of the participant. 

When you transcribe the interview, add a final section of your own field notes. 
Answer these questions: 

1 . Provide the details about the lesson on which the interview is based, including 
subject, time of day, segment number(s), and any other contextual information 
that would shed light on understanding the teacher’s responses. 

2. If you selected the lesson rather than the teacher, why did you select the 
specific lesson as a focus for the interview? 



Thank you, and interview concluded. 
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APPENDIX I 



Novice Teacher Study 
Codebook 





Category (Parent) 


Code 


Rules 


Exemplars 


1 . 


Administration 


ADM 


Any comment about administration, district, authority 
(e.g. relationship with teaching staff, attitudes about 
testing, etc.) 


The principal, the taught language arts last year. 
He’s helped me a lot with that. [NT07-003-R2: 9 Feb 
07] 

[A]nd when one of the teachers are absent here, 
many times--probably most of the time — they don’t 
bother getting a sub. [NT07-013-M1: 9 March 07] 


2. 


Classroom Context 


CC 


Any comment specifically related to classroom 
environment, context, etc. 


The first class just kind of...l wasn’t expecting so 
many kids to get pulled out in the middle like that. 
[NT07-019M1 & M2: 4 April 07] 


3. 


Content/Subject 

Matter 


CON 


Any comment related to the content. Does not 
include passing reference to math or reading. 


Some of the material that we’ve covered already, 
like Pythagorean Theorem and surface area and 
volume, [NT07-001-M2: 1 Dec 07] 

1 love math so if 1 put enthusiasm in it then they will 
catch on prettyquickly. [NT07-015-M1 ; 26 Dec 2006 


4. 


Curriculum 


CUR 


Framework for instruction and external expectations 
for learning (e.g. District Pacing Guide, Ohio 
Academic Content Standards, etc. 


It’s aligned with the national standards as opposed 
to Ohio standards. [NT07-002-M2: 31 Oct 06] 


5. 


Instruction 
[now includes 
differentiation, 
assessment and 
management] 


INS 


Any comments related to teacher’s instruction, 
instructional methodology, what was done in the 
observed lesson, what has been done in previous 
lessons, and/or hypothetical comments about what 
could be done. Indicates action as compared to 
orientation to action, which is Teacher 
Characteristics. 


We have been working a long this week on text 
connections and to teach 1 like to review the 
strategies and so we started with that just text 
connections. [NT07-033-R1 & R2: 12 Jan 07] 

So that was one thing 1 tried to clear up in the first 
class but made sure 1 explained it differently in the 
second class when 1 was actually up in front of 
them, just making decisions on how much time to 
spend on one thing or the other, where the 
problems were, what they already knew. [NT07-019- 
M1 & M2: 6 Dec 06] 


6. 


No Code 


NC 


For any turn with no associated codes. 


Is this the first time this year that they’ve seen that? 
Yes. [NT07-017-M1: ND] 
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Category (Parent) 


Code 


Rules 


Exemplars 


7. 


Parents 


PAR 


Any comments about parents (e.g. activity, access, 
interest, etc.) 


1 don’t use the same question as the achievement 
test because we do have a parent that comes in 
and does achievement test questions. [NT07-002- 
M1: 11 Dec 06] 


8. 


Planning 


PLAN 


Comments on planning (e.g. decisions that are made 
in advance of the lesson, how a specific lesson is 
composed, how materials were chosen or designed, 
etc.), including the question. 


It went, it went pretty much to plan. 1 guess with 
every lesson you just see how well they’re 
understanding it. [NT07-021-M3: 15 Dec 06] 


9. 


Professional Learning 
[includes collaboration 
and education in 
content area] 


PL 


Any comment about learning experiences, 
professional development, attitudes about 
professional learning, etc. Includes statements about 
background knowledge in content area. 


She was actually in my, my, and the person that 1 
student taught with last year. [NT07-021-M3: 15 
Dec 06] 


10. 


Resources 


RES 


What was used that day to teach the lesson (e.g. 
textbook, manipulatives, technology, teacher 
generated materials, etc.). Can refer to materials 
used to teach any lesson. 


Basically over the past 3 days, we’ve been, this is a 
3-day lesson we’ve been talking about percent and 
the previous day we actually got toilet paper rolls 
out and we had beans. [NT07-021-M3: 15 Dec 06] 


11. 


School Context 


SC 


Any comment related to the school profile (e.g. 
collective dispositions, attitudes, etc. or temporary 
conditions (snow days, etc.), physical environment, 
etc.) 


We have, we have a chart for the school building 
which tells me what standard I'm teaching what 
week and summarizing is what we're doing actually. 
[N07-012-M: 6 Nov 06] 


12. 


Standardized Testing 


ST 


Standardized testing comments (e.g. Ohio 
Achievement tests, TERRA Nova, district short-cycle 
quarterly assessments, etc.) Does not include 
assessment at classroom level embedded within 
instruction cycle. 


Well, then 1 realized that during their standardized 
tests, they’ll have a prime factorization problem and 
they couldn’t do it. [NT07-031-M1& M2: 1 Feb 07] 


13. 


Student 

Characteristics 


STC 


Any comments characterizing students: i.e. 
demographics, dispositions, attributes, student 
learning, learning styles, or reference to prior 
knowledge. Excludes student response to teacher 
action. 


So, 1 have students who are firecracker with math-- 
they get it and they go and they can be done, and 1 
have some that take a couple of days. They have to 
really marinate with it.[NT07-028-M1 : 4 Dec 06] 


14. 


Student Response 


STR 


Any comment about student response — cognitive, 
affective, behavioral--to teacher action. Includes 
student learning and student engagement. Refers to 
specific context. Not a general statement. 


1 find that if 1 assign them for homework or even 
during the class day using a pencil and paper, 1 
don’t have a lot of participation, we don’t accomplish 
a lot. [NT07-012-M: 14 March 07] 
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Category (Parent) 


Code 


Rules 


Exemplars 


15. 


Teacher 

Characteristics 


TC 


Comments about the teacher himself/herself (e.g. 
Identity, dispositions, attitudes, assumptions, 
affective response (surprise, annoyance, pleasure, 
etc.) Teacher as referent. Indicates orientation to 
action. Excludes observations about others (“1 think,’’ 
1 feel’’) 


It’s not bad enough for me to give them a demerit, 
but it’s just one of those things that’s really annoying 
to me. [NT07-002-M2: 23 Feb 07] 

1 thought it went much smoother. [NT07-028-M1 : 30 
March 07] 
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